
si OtO 797 



v&&'t& ; ' Sa-ti pcoetsslngi Statistical 



stttdlf* !0bt e^iplwii^r o£^\Lm m^^ij0 ^h^tlse availtbl€ fO£ data ba^t 
§#f0lppBin4^^nfl aaaly^i^t iM xili'Cl^^^if of tba stQdy to. 
i^atfltelMaSl f 03,1^7^ aiilog aa a al Jt loectfiii i Hh€ eciticalltf of atiob 

lllAsbaUcii^i laJtfttcitCmil d4i^&^$^&^ Siadf (ZDS) cavaala^ 
iao€&iplvt0Aei@s ^t«Mn^^ t^i 4a^a fatbt^td 1^ ^tha fl#ld and thjft- ft&ta 
fis^li;| ^in tVe i^.uA?, H^i^^f th^sa piroblaas vara tcacad to 

i yia<gi»tfa£Mi- f grg^ ^^ft^-tf ogld fc^y^ bf^tt" ^fi^g ^%td l i afaga h» dat^ itara 
axaSLT^i^* o£ ^ha lis, istii baa^ f aaiaa4Aatlo& ladioatt 

tki^ vpi^d^B^ m0 mtKtmi MIS of t^a 

ftato, pc^t^siB^ t4ela^ art aff'ott by^ 

t^ltflmtia^ 4p pcoblaas« T%im doeaatat bfiaflf 

'%aiartbM^tb# basiev aAi thi t^akal^al daaiga "of tha IDS 

ticj <h0 wthQmMmm osil ti Hhii iiiifeaniea of d 
pi etaayl#.s of. iKw^tt ii^in^ ^tffor-g^lving ar6 iscladad. 





* liproiflactlocs sn^ppli^fl by BDRS tr^ the best* that caij be nafle * 

• Mop th# 3ti ql:aai docaaent. ' . ' 




,TH(S tiOCUAAtM-f HAS ftCfW «6PR0-. 
-T»1£ Pi^lONOll <}ltGAMlZAT|OHOttlOlN« 

i gouCATiq^ POSITION ©R poticy^ 
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The rsftMmltotf^n a lufjor study datd ta^ t© verify thit tBe .d^f; \ \ ;^ 

' V . ' if " \ ^ V 

•ralynd are data that v^re cQllwt^ ^til.d isaiUy ^ vltMiid as . ' \ , 

qiufidvie. Howev^r^ wHw fmsais (^f natlQMl p^tley trs at *M#ka or wh^n the 
data bise Is *fO tafge co^pi^ as to liHr#t^# the I f k«l FhMd of error 

batweari data rteoriffig In tM ♦l^ld and tbt f ln§r aiialyiU data Biw 

' ■ ^ ' ' - ■ " ^ ■ ^ 

the time avallibV^ far phoetsilfis and Is sfioffef diti yerl- 

% ' ' ' ■ ■ '\ ■ \- ■ ' •■ 

f^catlon does bicome a -critically fitiportanr task., the IDS his stieh "pdlfey 

slgnffi€an&€ and did ftasult In a Jdrgn eanplM data bags, ^QfrtpMad In an 

'^inordinately short tMiai ^ ^ ^ ' ^ 

This paper describes XH^ j^Wamf nat-isn of the iOi data base is part of. 

( ' ' ' ' ' *■', • • ■ ' • . ' V ' •'- 

•n overal l iecondpry iralysl s Wfort. Specif tc ehiriges or eorrtctlooB J 

the IDS d«td ^Si as It asclsted at thu wd ef July are npt the fWln 

focus oft the>a%en It !^ h^pad that the reader ,v#i 11 aa^ln an ©xpMcit undtr^ 

standing of/tha Importam^ of this pHast jii any study, anJrislaht Into the 

naed for tffeGtfya >lfri/ilrif and .dWele^ewt of sdlt pri^ediires, and some 

overall guf del ineS for «ie in oth^r studies^ . ^ * 
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,€^^i^^^lh^ Campania tor V Edueatlm Study, a cm^rt- 

hM9tv# Nf^ r#SMi!^li proflrim condyettd In rpipDnie to a rondata by the lliS» 



."..AW ' ^ ^ ^ - . _ ■ ■ 

Coip9^^iftsm tn itfu^atlon Antridnients of 1974» Tfie IniplMentattofi of tM% 

stytfy took pUet frmi Mirkh IS^i through July 1977 and m% ednduet^ by a 

^nsortlynt hMd^ by Kfrsehntr Assaelstei^ Inc* (l^t) of VtesMngtarii D.tti 



KAI MS PespMsIbte for th€ ovarall conduct of the study , eel Ispted »[1 fitid' 
iMt%f and parfpfni^ ill eonputsr proeessfng of rav# data wfhteh rtiul t^ In tHe 
40$ data btse a^ailabja to the SMondary iralyfli affort« Olhar mwibers of this 
eonsortfum includadi 

^ • Edycatf on TURNI^Ey SystOTS, I ne / bf Washington, D,C. anayyils 
subcontraetoF to*KAI; 

• Learning Research and Davalopm^nt Center it ths Unlysrslty of 
Pittsburgh d8s!gn«and taehritear aisfitanet t^ KAI| and 

• Stetger, Fink and KDSSCoff, Irtc. of Ij&L^an, Virginia — assistance ^ 
to TURNKEY In tha Inttlal design and diVBlo^ent curr tculuni .inalyi Is 
proctduras used In ths study# 

IDS v^s designed and Implemented as a larga^scals In-depth asiessinent 
of the falatiorishlp between select^ Instructional acnstructi and studsnts 
DUtcomas w^th wphails or* aehfevmant (Cooley and Ltlnhardt^ 1975) / TNtia 
conft^cts are briefly daffned belawt ^ 

^ m Individualization Mtchtng of studants and curi'lculuni, tasting 
practical, asiIgniTTant and grouping practtcas, alterniitfve laarning 
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^6ve^1^^^ tet*«5en 3^hl#v«lsiK twts ^ whites taugtirV .jti 

• •. ^ s> •* y • »\ ' *• t^i. - ■ I '-^ , k • \ * \ ■ • •• . •= ... • "i " V .4.-, • . ' Hv • : * 

' ^\ ^ip Injtriicti^nal Events ©ia\iiiwi» niifiagwpfit vsv^ 1 

betevlofs ralatlvc t© th^ grfHip slM^ betng tuyghCp rfM thi nature \ / 

df Cue her Irttferac^lcris wltii stods^tsi wd . % rv. X ' 

• Tmc her Bac kg round— teaching ,«peri€nw» ferniiXa^ v*' 

recenijspecffle ^training* ^ ' . \ . ' 

In addltldfi te stydent outcomtf dati and data on t^se f W;t cp^s^Wcts,^ 

* Infcrnjatton on Inltructional stttlngs f^r ccmpOTiatbry edyeatronp eosts^ 

services provfdea by ^^ehool^ dUtricts, Vnd_ th€ criteria used.'|^. districts , 

tfr^etennina crttipansator^ tl^tglbllf ty fnd.partlclp^tldn for fcheol 

btfildmls and students wm. gathered, [ - \ 

thf.abal^ils^ sa.mplks weVp obttlrted froffi ^ Scf*^! tulldtngs irv 

schools 'd fit rl<;ts; with :approxrrimta|y »A00 Slrss^o]^ tga^htrs^ '23tf fcorapen-. ^ 

satory educatron ttat**ers^-ifld U,000 studints. ■ Prsteits of^studant achieve- 

mmt and attitudes were'ecndufited'^ In Septrttbar and elrjy Octobar igT'fi, .and : * 

pQStt^sts Wara adjiiinistered fn April and airly May r9/7^^^n tHe fall of f977 

foMovp^up tat^tfng vms conduttnd'^ Mr a satn^la ofnTos stwj#i|its *b 

eicamlnf changas in achiav^tfriant^sec^s over the i«nmsr 1977*. 
^ . = ' ' - * ft ■ 

The IDS secondary anil ysl f'af fort being cgndyjl ted by TURNKEY iinca Aufust 

1977 an90mpasses a sirlas pf ralatad tasks desfgnad to Explicate ralationihi ps 

within the l&^S data b#i5 more full^ that wis poisibla djjring the primary phai^ 

Ortglnariy Fntendad as §r? Initial- tasR T^a^lng to further analyses of n . 

■ ■ ■■' "-^ ' ■ ■ ^ ■ ■ y X J-'- ' • ^ ' " 

■ - . . ' ■" . N . . ). ■ . . 
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• i'^' ^^icl*lc .Issues rntended for further study ^c»ud«ds aar^ detailed 

' • stu^^'^f iM^fetloSt^ttlrtf » relating groupinf heterofweftif 

:-'V -r :•• ■•*:•>" -J^s'- v\*' ■ ' . 

" ' ' 'euttM^s, I^Bitlng •cteracterlstlcs of effective tMcher^, •x^lioi'lfii 

."i Z e^dJtlorwl slWent V Prodess oatcomes, studying the Inftuenc© of «M»ysls 

"rtietheds ©riVstudy findlnis to date, ar^d developfng alternate ino^l s for 

predlctLon; of outcomes. j 

The »lie and complexity of the IDS data base and the sever Ity of some 

. of Its drta problems heve caused this Inltlp^ task to require much mere 

attention than was origlnany envisioned. The scope of this task and Its 

Impact on ilready published prel Iwlnary IDS results became the Impetus for 
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IxhlfeU t pjovldM a tchtmtU flew ehirt of IDS student. ^r©€fSSt .f^^_ 
MnA other fi1ss« A total of 20 MgniTtfe computar tapas €Mti1nirig ] 
rwords for all ftUs ihOMn in Exhibit i accept thosa ralat^ M ttia > 
Gufflculiiii Anilyili Ovarlap and the Genarar CuikIcuImi Ana\ysl^ Suamiry 
wara traniHrrmi to TURMKEY by KAI at the beglhning of togtipt 1977. All) 
raw data documafits (test t^okletSt teacher Iriterviews, ate.) i^rs also 
tfansfarrM to TURNKEY at that tlme« The two eurriculum analyiis ft las 
mantlorted abovs r^nain In storage on KAI data^fik storage iintti» slnee 
their storags method precludes easy ^transfer to tape« All v^ldMCipai have 
been stored at TURNKEY slnee the spring 0^977. - ' 

IDS student data flies include studejit^^|pi1 data on: rsgiifar and 
supplmantal Instructor, time spent In^regular^ftd slipplemerital irv^triictionp 
, attendance, compensatory <educat ton statuSf €ttinie (pre and post)» number 
of itOTs in pcsttest overlapped with curricular exper fence achfevment 
(pre and poit) ^y subjeqt subtest and total subject* The Roster File groups 
students as thay were tested; the Student Master File groupi students by 
regular Instructor and (for each regular instructor) by suppl^sntal initructor 
The Student Master File serves to define both analysis units used for the 
study: ^^ss room and Instructional unit. 

The classroorn analysis unit Includes student data from a1 1 students having 
the same regular instructor* Process data for the Instructor are brought^* 
together with the student data to complete the classroom data flies. 
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District Pipgrin) Riport 
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^ - r^guTer Instfuettr and a eanwiDf! suppl mental Instru^J^r, Proatsf dgtii far 
* thti* two frtstryeters (eM-regular and onm %upp^wmn%p]) are j^blA^ usihg 
i Weighting prM^ure and mtrg^d with tha student data to dompl^tt the . , 

^ ins truetfonal unit data fllas, ^ 

. ■ ■ . ' ' • 1 . ' ' . ■ ■ ■ . 

IDi procasi data fllss Includa Itams covering all aspeets of the proeess 
canitrueti dascrfbad briefly fn ths Background section. The fburces for thsie 
data Include: teachtr intervlewip studertt asslgrenent records, yldaetapes of 
classroom experieneeSp curriculum information (both clasifooni ind studw 
IsveOf and estlittates of test/curr iculum overlap made ty the teachers at th^ 
tlM of posttasts in the 'ipring of 1977* . , 

Other IDS data include schbol fcnd district level Interviews, raportSp 
bydgat documents, prograni planning documents, and ether Jnfonnitiofi relevant. 
to the Issues of compeniatory education allglMlIty and partlclpahts, profraBi 
€Osts» and district poverty/wealth statist IcSp 

Exhibit I clearly d©nionstrat«s the cortplexity of the overill IDS data 
base. The discysslon in the Background section Indicates ttie large size of . 
the dita base and undericores the nat ional. pol Icy Importance of the issues 
addressed by IDS. Since all final processing of the key analys Iff i les 
had to be conpjeted between poittiiting which ended In May 1977 and tRe end 
of July 1977, all three elements outlined In the Introductron are present Jo 
Justify some sizeable effort to verify that the data analyses reFwrtid fay 
the IDS study team (Brady, Clinton, et at, 1977) and by NIE (M IE, ,1977) 
were Indeed based upon the data actual 1 y col lected, . ^ 



■ • ■ , ■" \ 

I nit la \ Re exafn t ya t Ipn - 

Th€ Wsfc int««lon o-F Int tli! reexamination actlvl tl€s. uftdwtikftn as 

^ ' * - . . " ■ 

part of tNt IOS*se«ondary aniiysis m's to compare data rts id I ng in-Ust- 

stage'Cjust prtor to snilysls') data files to raw. data docufnents Iff order to 

Vtrf fy that the da ta analyzed corresponded with the data col lected in. the 

fle1d„ rroffl E5chlblt lit can be seen that the Instructional unit data file 

f ~ 

|s_the tventiJal re<ep-tible for all process data after coding, keypunching, 
(nerplng, sorting, combining, we ight ing, etc. have been, accompl 1 shed. Actu- 
ally, further^d€ta prccess t ng 'for analysis purposes vas dpne .using the 
Instructional, unit di*a fi le, but froin that file onward the foriri of the dat,a 
(coriverted to s tan ^ird z-scores and summed to create sub-construct scores) 
and the cofflpleteness ef the data (some items dropped due to fnissing cases, 
etc.) v^uld Nve resulted In a data verification process both more difficult 
to acconipl Isfi and les = tornplete than the comparison points chosen. 

fhe Instructl ona 1 uni t data f lies contain approximately 240 student and 
process msasijr&s f or *dch unit. The student and process data flleiof 
Exhrbit I provide ih« icores fof each of these Instructional unit measures; 
Che positioning of l),^^<^ 2^.0 measures in the instructional unit data file is 
according U) 4.wH.ir.i.t and (b) student data type. IDS instruc- 

tiorwl mii J-u. rll^^— J-l- 334 units for first grade reading 
(R1>, Ziy fw. ri,^iy.=.J= ..lail. (Ml), jbZ for third grade reading (R3). and 
289 for tMlr*l ■^ikSs nwiiii if^i) 

For pur.pa*ei i^i *i«u we, I i ,„ai.u.n a y» tenuna t ic random sampling cechnlques 
were used to i „ Si ...b idr.ipU from each of these four data files 



(Rl, Ml, R3, M3). The instructional units within their respectl\/e files were 
arrange in asGanding order (by instruct ioriil unit Idtnt if icatlon numbsr} ind ; 

Identified by nuniber (from 1 to n, whers n Is the ntifriber of instructional 
units In tach fJle), Using a randc^ number table, m unique starting point 
for each sample ^las selected from the f if%t twenty recordi, Ut ing the starting 
point as the first of the units to be Includid In the data verification 
sample, every twentieth record thereafter also mcluded* A total of 1? 
ynits were thereby identified from the R1 file; 12, from rtl ; 18 , from R3» 
and 1 5* from MS * 

Using available raw data documents, and deflnitioni for the meaiures of 

the instructional unit data files, instructional unit measures vvere caleulitit 

and the results compared to the data value resident in the instruationt 1 unit 

data f f le for that meaiuie. Error bounds used were, t 11 of |he maximum 

score for fneaaurteb hav iriy a fixed iria^inium value, and ^ 2^ of the full file 

average for inaa^uitis havirsy riu fixed maxImLjfTt value. Choice of this specific 

deflnitiQn uf arr^r Laund ^ gibitr^ry, but Intended to be fairly stringent. 

Not showf) e^^pli^lLiy in txriibiL I the fact that the last-stage data fi le 

vff.ere indlvidvial .luJ^nL^. in^y be Identified f or ver 1 f I cat Ion purpos a| 

is the Studenl H^.L.i MU L,inwe fi^iid »c&r I fig of stiiJeiiL test bookleLs vvas U sed 

for varificaLiuci of bLuJeiit ^careb. It was felt that rescoring of all test book- 
\ _ 

Ut^ for ep^N I ..^i, L Iwu^ I u,.lt In Lt>^ ^/ ^\ ifl^atfufi sampl 6 wOU ^ be too cos^tl y. 

Tfiua, o Hi«*- ' I I- ...1: - -io»Hi-J wl... , u.ite dO Initructlonal Unit 

v<ab seUwLtJ r , If,. ,-11 n Uai lu,. saiiiH lo , a bfli.iMleuf 2 ui 3 students 

witfiin ihwi I u.w,] r,.. ^la^oi aw^., = vei I f i cat Jon . The coiripar I son po f nt s. 
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for itudent . scares wsre nn4 scprtd results from'test booklets verius Student ' 
Nastep File data values, Ach f evtsment pr-e^cst booklets were not available, 
glnei these ha^ beefi r eturn«d to the particlMt'ng districts. However, card 
Imagfe records of th« 1 terti P«sponSes for these pretests were avail able., and ^ 
these were used In place of tl^e bookl ets t hern selves, thereby llmitina the 
ais*ssiDent of pretest file daCi ^al I'd I ty to keypunch accuracy at best (not a • 
dibllitating 1 imftation since observed keypunch accuracy throughout the IDS 
data f ties Ns been up to accepted con»nerclal standards). Error bounds for 
student scores were t 1 test IterTi. Again, 'choice of this definition of error 
bound -WIS a rbi trary but 1nt,<rided to be str Ing en# 

Results of these in1ti4l reexaml rBClon act 1 V 1 1 i es Indicated a substantial 
numfcer of p roc ess ni«a*ur«5 meaisu res) having errors in 51 or more of 

chelr verification «..»Hl« values . Addi clonal ly , student score verificatiori 
activities indicaie^ t ha t « ppr imate ly 20S of the third grade students examined 
rromthe vfefiFitdU^M w.npU Mad errorf'in posttest achievement scores in 
excess of t 1 L*,L Hen. u^jund, k\,a, It was found that 6 to 12% of all 
students haJeH^i* | ^ f eL „ c s 11 1 tudeS scores in excess of the t 1 test Item ' 
bourd. h in«lly, Ik . r..u,.J what m of the Ml students exam^t.ed fur 
the yttrlflw^ilo,, proc«*s f« u ^rr ^neou s tfompensa tory education status data. 
PI, SI a.-J^ ,^hl^.^,„«,n »...u^sts, d 1 ^a, ievement pretests, all att i tude po St - 
test.. a„J .11 ....^..„^->^. . ..i..., .1 1.1. .tatu. ddtd except for th= Ml file were 



p^fllad Reixanii lotion ^ . 

^ Onct the Initial reeMffli nat ion was cornpleted it was apparpnt that the ^ 
scape of the indicated problems required a greater scrutiny. Thus, the 
first result of the initial reexamination was the realization by TURNEKY 
'and mi that reanocation of resources away from desired analysis taslts to . 
the detaned reexamination of data would be necessary. 

It was also clear that in order to preserve as much of the current 
itudv's available resources for anal yses , it would be rwcessary to Kloritize 
.the problem areas in some manner. A plan was adopted for assigni ngjesource^ 
priorities that included the following steps: 

1. Focus only upon iho^e process measures or student data areas 
identified \n th« initial reexanii nat Ion ; 

Fur e.ch of U.e.. m«sufes, view the units in the verification 
.a,»HU in tww dl.Linct categories: (a) those whasB values fail 
witnin th« e. .w, bwund, and (b) those whose values do not; . 
^ ct.= lb) ..i.yw.y Jeternili.e the average size and direction 

(«l^=b..i. -lyn) xh^ ubserved' errors and deterinine the average 

U...« ...ulL. Lhe full file byn,«ans of the following 
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E^* (1-P)(es) and Ea ^ (1-P) (elT where . 
V P - prQpdr'tTon of the verlf Icition ^pU , fal^l ins Intq , ,, ^ 

category (a); l.«. , hav Ing.'acceptabl e •data* ' . 

' ' • €5 - averagt algebraic error size f of ^category M units ' ' 
eg . avftfage absol ute. error size for Mtegora' (b) units 
Es ^ projected full file average algebraic error • ' . 
la ^ pfojectei full file average abi^ute error; ^ 
^. .5. In order to compare projected full file errors across m^sures ■ 
having basically different scales, express the Is and Eg values 
so calculated as a percent of the maximum scale values fcr those 
measures havlpg such a fixed maxlmuni or as a percent of the 
average f u 1 1 f 1 1 e velu* for measures haying no. fixed fnaximum; and 
6. Rank order the measures accord ing to the resul ti ng percentages . 
These 'steps can best be surmarized by saying t^,at the priorities for , 
attention^ should go toward those pleasures whose observed errors v«uld %ave 
the largest ^mpact on analysis, where impact is expressed as a percent 

of ioma SGale value, 

An illustration .ay b. t.l pful here. Suppose B0% of verification sample 
units Indicate ac..H..UU .a lu^s for Measure A, 60I for Measure B. Measure A 
hasa fi^ed n«.i.nu,„ va lu* of 1,00; Measure B has no fixed niaximum. but 

has an av«r.ye .cU ..iu..f 75 for the full file fro^ which the verification 
sample was drawn, T„„ average algebraic error size for the 201 of the sample 

. . .-r ki^w^aIlih^ Is -ho for Measure A; for Measure this 

units showing una^weptabl e v^aiues is lu i 

average Is +5. ^ha av^r^g. absolyte error ^size for^^the units with errors 
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Is 15 for 'Measure A» 10 for Measure B, Therii 

Meyfturg A. , Mtamrs B 

^Cs^^Cl-'SO) (4=10) ^ +2 ' Is ^ (l-,60) (+5) ^4-2 

- (lr,&0){l5) - 3 - (1-,60)(10) - k 

(€/l6o)^ X 100^ 42,dl CEs/75) k 100 ^ 4-2.7? 

* (Ea/lOO) X 100^ 3* q« ^ (Ea^7fe) )c 100 ^ 5JI 

In this Illustrations Measure B would get priority for cDrrection oyer 
Haasura A since rank ordek^lng of the reiulting parcentagBi showi B in 
greater need than A whether the fQCus is one Eg or Eg, 

One additional step is needad: application of a cut-off percentage 
for the values resulting from step 5. IDS secondarv analys'is efforts haye 
used an arbitrarily chosen value of k% for measures having a fixed maxlnium 
scale value and 2% for measLjras having no fl)c€d max ifnurTi. Also, it ,has been 
the case to consider a measure as being a candidate for correction if 
either the Es-based or the more stringent E^^based pereencage exceeded the 
abcya cut-off valuer, ^ s 

The above described steps^ of assigning prloritv for resource allocation 
have been appl iad lu d 1 ! process rnaasures. As a result a core of 32 fneasures 
(of the in inltiaMy iJciitified) dre befng subjected to a correction process. 

Due t^ the critical nature of the student d^ta involved (compansa tor^ 
education status, pabtte^t scores), no pr i or! ty ass ig nments were made.for 
these variables. Inbtead dll errors were studied. All sources of errors 
were identified and correcLiuns were made for all major errors except the 
attitude measure pretests. Nl£ felt no priority should be assigned to 




^correcting the a tt 1 tude measures. 
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The task of idintlfying the source of the error has been carried 6ut 
In iir^ses by tracing the data flow ctiart (1 ike that shown In fxhTbJt }) ' 
fcr thi point at whith the ©rror seems to enter -the flow. It is then ^ 
^^jttf mined If the error and/or its entry point into the data base is 
syst&nmat Ic'p and program porreQtions from the last point of correct dita ^ 
are initltuted/ No hard and fast rules can be offered on these steps iinca 
each study's data files have their own unique hiitory./ But the^ logic in- 
herent in their deyelopmant will provide the most pov/^ful clues to the 
source of the probl ems , and , therefore, to the correction steps neeWa. 

Short histories of three data probl em s ituat ions wl 11 be presented so 
as to iUustratfe the conduct of the above tasks In th*,JDS setoodary analysis. 

The first situation Involves the third grade posttests referenced in 
the Initial Reexamination section. Recall that^ZOI of the third grade students 
examined from the N/^r i f i ca 1 1 on kample had errors in posttest achievement scores 
in tpccass of the +1 te^t i t©n bound . In fact, these unacceptable cases had 
scoras in the Student Master File averaging 21 3 points below the hand scored 
test results obLai'neJ frwm Lhese students* actual test booklets. The magnitude 
□ f thi^ I u.^ Lecame th^ iniMieJidLe focus in the search for the source^of the 
error. ^Wf.ai ^wuld causc a wt.Md's score to be recorded as only half of wha t ^ 

I t d^tUal ly 

rhe answer y.^^ rv..nJ h. Uie H'---^^ f w 1 1 uwfcd i f» obta In i ng a test score once 

the bookUts we. t f.^eiv^j r.um uhe field. Tests were not scored by hand In IDS; 

rather, the child's r^^pon^e lu eacf, quest lun was precoded and keypunched. 



'i'J 
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These responses, recorded on keypunched cards, were then scored via a computer 
program, the responses made by the first grade chHdren pre erd, post and - . 
third grade chi Idren at pretest were identified in the test booklBt by the 
number (e.g.; 1. 2. 3 or 1. 2. 3. ^) correspond ing to 'the selectad answer 
Jor each questron.. However, third grade posttest responsej^we^e either 
numbered, U 2," 3. ^. or lettered. A. B. C.-D. for each odd numbered test 
question in a subtest, but the even numbered questions used the numbers, 
5,. 6, 7. S, or the letters, E, F, G, H. All keypunched responses were 
coded as numbers with the codes being assigned as follows: 

THIRD GRADE 

CHILD'S RESPONSE KEYPUNCH CODES i ' 



1 or A ' 

odd-numbered 2 or B 

questions 3 ^ ^ % 



5 
6 



> ^1 u 

questions 7 or G 7 

8 or H 8 

B.r... U.I. .J ....^ U. K.y^ «1I L«l bookleis were coded onto 80 = 

.o,.„,n reco,d l.y.wts w,.,. ir . certain coder u«d the conversion^ for odd^ 

numbered question, for the «ven^nymbered rtems onj the third grade po.ttest 

,n ir.^.. n.. .... my .Igorithn* wou i d eo «.,eou s 1 y Indicate that 

all aven-nJr;be..d qu..t,,.,.. r.ad .e.n answered uncorrectly. The unacceptable 

cases were e^..nined In llyh. thi. hypothesis, and the hypothesis proved 

correct. Codlny .h.«i. fo, cases verified the erroneous coding entries 
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And since all coding sheets for third grade p^sttests were ayailable., project 
staff were able to Isolate all cases of this problem, ' Fortunatsly the sample 
used in the fnltltal data base reexamination proved unrepresentiti ve of the 
total IDS sample; only 10 classrooms out of 200 were actually involved. Test 



scores for these chndren werg corrected, thereby eliminating the only major 
error found related to achfevament test scoreis. This si tuatfon coul d have 
been avoided by more thorough edit procedures* 

The next^ situation involves the Ml compensatory education status data 
also referenced earlier. Recall that 2g| of the Ml students were foijrid to 
have errors^ In their status as Indlcatgd in the. Student Mister FIW. Befortf 
the initial reexamination had revealed this errorp questioni had been raised 
within the study team over the large number of students asslgnid a compen^ 
satory education status In the Ml file. The number of such students In the 
Ml file ma^ch^d very closely the number of Rl compensatory education students* 
NIE's report an Campenbd tor y Education Services (July 31 p 1977) Indicated that 
nationally only about half as many students participated in ^^CHnpensatory math 
programs as do in ^u^mpensa tur y r ead 1 ng progranii. This ratio v^s evidenced 
at the thlfJ yi aJe , l^aviny l he first grade data in even stronger dotibt. What 
if a sLuJ^iiL ^ , *..MrdeJ ^t«4..^ ii* Lhe Ml file wa b nut a niath staLus at all but 
Inst^d 1,1^ fisi »Mdiny wufupeiis^tury cduwatlon btatus? Thib hypothesis 

would ^MpUin u.^ wiiL wv^, ^LuitJariwc uumpensatory ^ducatiori students in 

the Ml file IhU inJ^eJ ih& yroblmu. When cfMtlng the Ml student f i Us 



in t^ verif Icacion saniHle fi^d correct data for compensatory status; 711 
happened to Ymve a Ml bLatus Uiat In fact agreed with their Rl status, This 
error also was correcLed, 




entered. None of the Ml students 
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The final |ituation fnyolyes a set of process trrors identffr&d aftir 

thte applicat Ion of the six steps described earlier In this sectfon. All of^ 
^^ the procsss insasures derived from the student aislgnmtnt docLunents (Todiy'i 
Assignment. Sheits or TAS) were found to have eKcessive error rates* Using 
the logical flow* chirt for thesa data indicated one source for this problem 
very r^ipidly: whenever a teacher was In the study for reading as well is for 
^.fnath Nstructlon, the TAS data for/ reading was not only cdrried Into the instruct 
• tlonal unit data files for reading but for math as wall another exampls'^ 
of a process ing error. However , while tracing specific additional 
errors Identified for individual teachers, ieveral aases of dupl tcata TAS 
recordi v/ere found In the TAS data file; i,e., two recotds wl th di ff er ing 
data but the same identical teacher id#fft 1 f icat Ion number. Following this 
lead revealed thu nUarly 10% of the TAS records were, In fact, duplicates. 
T>ie moft conrnon pattern fur thesa duplicate records is one record with the 
eorrect Md^t I f i c^t ion and correct TAS data plus one or more records with ^ 

the same identification but blank TAS data, A current hypothesis, not yet / 

/ 

verified, Is that lAS raw data documents which involved more than one sheet 
of paper rmy fi^w^e bee., . lii bume instances, erroneously coded as one record for 
each sheet. Tfjcbfi. muin M^ge instruments had the Indent i f icat Ion code 
on each sheet But f.ad the TAS data values written only on the top sheet, 
It i& pa^sslblc Lii^L Id ^uifie cdSes Lhese multi^page TAS documents were podad . 
" as If each sheet weis ^ separate record. Corrections to, these TAS data 
are In progress 
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A&tfvlties PevetQping from Data Reexamination # 

Corrections to student diti trav© been completed and the Student 
Hasttr FllM have been rebuilt and restructured ^ Include additionil keys 
for tving IDS student level data to all aspects of the IDS data base, 
additiDnal overlap data, fend revised student-level instruct ioha 1 time data ^ 
whioh reflect actual student ittendance, 

CDrrections of the key process errors identified are in progresi. For 
some a ipecific spurce has not been found so reprocessing will be done from 
Che raw data level. For others sources have'becn located and the corrections 
/ are bel ng made, 

/ The impact of the corrections in student data is dramatic In some 

eases. As an illustration, Exhibit II Is taken from the NlE report of 
Septeniber 30, 1977 (The Ef facts of Services on Student Development). This 
exhibit presantb f d 1 I dnd spring achievement test scores for compensatory 
education iLuJentb in the uveral! study sample (as printed on p. 20 of the 

Nil report). Exhibit MI presents the^ revised results corresponding to 

/ 

Exhibit II bdbed ufi the currecLed student data. / 

^ / 

C(?fiipdi i iiy t/siiiLiL ii and Ml shiQw^ that the large g^iins for IDS cornpen- 

/ 

s^tory education wtiMd,toh lefei^nued in the NlE repor^^are actually even larger 
than or igl na M y Lri^uyht. Ii ^fiould be noted p^at conges In the sample sizes 
at all yi dde U^/4.u ^ic brouyfa ^u^uL by the corrections referenced above and 
by sorriG addlLiMnal wHip^ii^d lui y eJucaiion b ta tu/ cor rec t ionS identified as a 

by-product ut tfte .oiie^LiiJn Hlvjceb^ itSelf. 
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FALL AND SPRING ACHIEVliiENT TEST SCORES FOR 
COHPIMSATOfrY EOUCAt/oN CHILDREN IN THE 
IDS SAMPLE (FROM NIE REPORT)* 

(Mean) Raw Score 



1 


n 


Fall 


Spr i nc[ 


Gain 




1,355 


.23.8 




23.6 


Hhthamatfcs 1 




15.6 


27,3 


11.7 


Rgadlng 3 




■ 19.7 


31.0 


11,3 


Mathtmatlcs 3 


792 


21,3 


36J 


15,4 



i 



*Nttionil Institute of Education, The Effects of Services on Student Developnpent , 
Washington, D.C, ; UiS. Department of Health, Education, and Welfare, 
Septeniber 30, 1977- 
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Exhibit III. 



FACL AND SPRING ACHIIVEMINT TE|T SCORES 

m mmtmtm foucATroN cHtLOREJutN 

THE IDS SAMPLE (REVISED RESULTS)* 



(Mean) Raw Scores 

n Fal 1 Spr Ing Sa In 

Reading 1 1,415 23.8 kl ,h . 23,7 

Mathanitlcs*l 630 " 15,5 27U 11.1 

Reading 3 US^2 19.7 31.7 12,0 

Mathematics 3 830 21.4 38.6 17,2 



Ms of 3714/78; bastd on work performed under contract No, NIE 4O0-77^0OS5 
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Exhibit \Mp taken from the NIE report, compares the achisvem^nt of 

' ■ ' '' ' ' ' ■ ' ' _ ^ ' li. . _ 

students receiving instruction in mainstream programs to those f rem pyllout 

# 

programs (as printed on p. 23 of that-^port). Exhlblit V presents the 
ravisad results correspond i ng to Exhibit IV. 

Comparing Exhibits IV and V shows an even more dramatic effect of thi 
student data corrections. In its report Nil Indicated that neither settings 
(malnitream vs. puHout) appeared to be consistently associated with greater 
Instructional effectiveness. Exhibit V does not support thl s assert ion ; 
mainstream gains are significantly greater In three of the four ceinpar iions 
with no significant difference In the fourth. 



Reading 1 



Hathifnatfes 1 



Mathenatlcs 3 



eONNNSAtORV EDUeATION CHILDRIN RieilVING 
SifTINfil (FROM Nil R|PORT)* 



(Mean) Raw Scores 

ierrwa -Wn 



istreain 311 2*.!^ S1.3 27.0 

fullout 1,0^ 2ii» > ^6.3 22.7 



istrtaffl 1J72 15.7 27,9 12;I 

Pulleut 2S7 15.6 24, i» 8.8 



Readint 3 

I 

Pullout ' 1i211 19.8 31.2 11.4 



Mainstreafn 195 19.4 29.4 .10.0 



MalnstPiiin 170 20.5 32.5 12.0 

Pullout 622 21,5 ' ihS 16.3 



*Natlenal Institute of Education, The Effsets of Services on Studsnt Dtvelopment 
Washington, D.C.i U.S. Dipirtrntnt of Heilth, Educition, and Welfare, 
Siptember 30, 1977. . 
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. Exhibit V 

FALl AND IMING ACHIEVEMENT TEST SCORES FOR 

w- ' . ■ '•- •■■>■■ ■ ■ ... ... 

^^iMMPENSATORy iOUCATION CHILDREN lECElVING INSTRUCTrtfN 
I PUUOUT OR MAINSTREAM SETTINGS (REVISED rttSULTS)* 



I 



.Mainstream/ 
Puttldut 

Hathmatles Grads 1 

Hatnstream 
Putlout 

Readfng Grads 3 



Mainstream 
Put lout 

Hath@natlcs Grade J 

Ma f nstrMm 
Pullout 



n 

309 



307 
323 



198 



165 
665 



(Mean) Raw Score 

Spring 



23.7 



15.ft 
15.6 



19.* 
19.8 



20.6 
21 .6 



51. i» 
% ^»6.5 



29.1 
25,7 



34.3 
31.3 



37.7 
38.8 



27.1 
22.8 



13.6 
10.1 



14.9 
11.5 



17.1 
17.3 



*As of 3/lV78f based on work performed under contract No, Nil i»a0-77-«0065 
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eampl^ity of the study, the time avail able for data base deVelpiMMi^t and 
aralysISt and the relationship of the study to '^UMtlonal po^^ ma 
^1 Iritertgse ttie trltleal Ity of ^ch reiMtimr nation. A^gentertrt*^^^ 
sayi before bellevlnf to© strongly In the lndlcat|^ « of a sfMd 
look baek and ask If the data anal yied correspond With the data cbneetad In 
the fJeldi j Understand and use the logical d%ta f/ow patterns of the study 
In this reeKamlriatlon. / \.J^ 

Draw from all available sources, Tht'^f^seebrtaery aw^ 
Includ^rtoJh^rganlzatlon^ (KAI and TURNKIY) Inv^ved In the Implementation 
phase of the study. Only through the combined effort of all study personhel 
have the problems been Identified and, most Important , so1v#d. 

Hold onto as much time as possible for the data processing and analysis 
phase of the study, IDS and Nil time constraints left less tim# for these 
tasks than. Is normally desirable. But, whatever time is avallabla 
should be effectively uttUied by rel ianee on pre-tested and thoroughly 
applied edit roi,||||pas. 

Be prepared to prioritize data examination/correction efforts if more 
problems are found in the "'look back"' thiin anticipated, 

Overall, remember that neither elegant statistical techniques nor a 
strong analysis n^el can overcome erroneous data. 
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